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A SCALABLE COMPUTER SYSTEM 

FIELD OF THE INVENTION 

This invention relates to computer systems and more particularly, but not 
5 exclusively, to a computer system which is required to have a large number of 
computer processors for use in large scale system applications. The present 
invention has application to scalable telecommunications and intelligent networks. 
BACKGROUND ART 

Recent developments in telecommunications and intelligent networks, 
10 primarily involving the field of photonics, are resulting in a rapid expansion of 
bandwidths available for communication. The available bandwidth is currently 
growing at a rate involving, roughly, a factor of two every two years, and it is 
anticipated that communication bandwidths may increase by at least three orders 
of magnitude over the next ten years. 
15 in order to match this rapid bandwidth growth in the level of 

telecommunications, equivalent computing power is required. It is therefore 
desirable to design a computer based system architecture that is massively 
scalable. The scalability is essentially driven by the number of independent users, 
as in mobile phone devices, rather than the complexity or size of an individual 
20 application. 

Hitherto, there has been no known solution to the problem of designing a 
massively scalable architecture for the intelligent network (IN) domain. 

The Internet is a large computer system, but it is a hierarchical system and 
hence does not address the issue of massive scalability. 
25 Other computer system architectures that are known are only scalable in 

many orders of magnitude less than what will be required in the near future in 
telecommunications and intelligent networks. 

For example, the computer system of Figure 1 is based on an Erlang/Open 
Telecom Platform (OTP) running on a single node. The computer system 10 
30 includes hardware 12, an operating system 14, a display 16 and a keyboard 18 
and a suite of programs 20 which include application programs 22 (eg. in 
programming languages, Erlang and C), sourced programs 24, run-time programs 



WO 01/02972 



2 



PCT/AU00/00796 



26, a library 28, and a database 30. The system may be linked to an external 
database 32 if required. 

The single node provides reasonable system development facilities, 
including an Erlang real time environment, or interpretive environment. However, 
5 this is achieved at the expense of potential performance owing to the 
interpreter/operating system layers. 

The single node computer system of Figure 1 may be linked to other similar 
nodes by an asynchronous transfer node (ATM) switch, such as the AXD-301 
switch with satisfactory performance. However, this switch has a scalability of 
10 1:30, which is orders of magnitude less than that which is required for ultra high 
u communication bandwidths. 

Sj Referring to Figure 2, there is shown a split node computer system in which 

W an OTP node 34 is split into two closely couple nodes: a COTS (commodity of the 
shelf) system 40 and a multi-processor (MP) Erlang Engine 50. The COTS 
31 15 system is essentially the base system and may comprise of a UNIX operating 
: platform 41 , application programs (eg. in C, C++, Java and Erlang) 42, a disc drive 

fgss 

m 43, graphics 44, an Internet modem 45, an Internet interface (TCP/IP) 46 and an 
JJ( input/output interface (I/O) 47 for communicating with the Erlang Engine 50. 
p The Erlang Engine 50 is a shared memory MP system running software 52 

20 and 54 in Erlang on top of an optimized message passing kernel (56) such as 
QNX. The Erlang Engine 50 also has an I/O interface 58 for communicating with 
the COTS system 40. One processor of the MP set can be devoted to monitoring 
software, the remainder to functional processing. 

The split node OTP system 34 of Figure 2 may form part of a network that 
25 includes a plurality of regional processors (RP) 61 and support processors (SP) 62 
for operators. The regional and support processors 61, 62 are connected to 
central processors CP A 63 and CP B 64 and to each other by a high speed RP 
bus 65. The MP Erlang Engine 50 includes a high speed interface 59 for 
communicating with the central processors CP A 63 and CP B 64. 
30 The split node OTP system 34 can be linked to other computer systems by 

a switch 70 such as an AXE-10. For this purpose an AXE programming system 
(APS) 72 may be provided. In a telecommunications application, the interface 
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shown to the AXE-10 may be implemented as a high speed Ethernet, primarily 
due to the availability of the Ethernet PLEX (Programming Language for 
Exchanges) blocks existing on AXE-10. The Erlang to PLEX interface has been 
demonstrated in two modes, firstly with the AXE-10 controlling the links, as in a 
5 call forwarding application. The second mode is with OTP in control, with an 
application such as remote changes to AXE-10 tariff tables. 

The scalability limit of the Erlang Engine is a maximum of eight processors, 
given that it is a shared memory environment. The eight processors, together with 
a demonstrated 5 x speed up from the move to compiled code, plus 2 x moving 
10 from Unix to QNX gives a scale-up of 80 from the base system. 

Therefore it is apparent that both of these known systems have limitations 
in terms of their scalability and could not be considered massively scalable. 

It is therefore desirable to provide a massively scalable computer system 
including a large number of processors in which each processor can communicate 
flj 15 effectively with other processors without regard to their locations, 
u SUMMARY OF THE INVENTION 

fU According to one aspect the present invention provides a 

03 telecommunications system having a plurality of nodes, such that the plurality of 
nodes are clustered in a plurality of interconnected neighborhoods, the system 
20 characterised in that a relatively small number of cross-links are provided between 
nodes of different neighborhoods. 

According to another aspect, the present invention provides a large scale 
computer system including a multiplicity of nodes, each node having a plurality of 
interconnected processors, said nodes being arranged in a network with 
25 neighbouring sets of nodes of the network forming neighbourhoods of fully 
interconnected nodes, wherein random links are provided between nodes of 
different neighbourhoods in the network whereby each processor of the system 
can communicate effectively with other processors regardless of their location in 
the network without full connectivity in the network. 
30 In essence, the present invention has come about through the realisation 

that problems associated with telecommunication and intelligent network 
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scalability and/or computer architecture design, can be addressed by applying the 
"small world" principle. 

The "small world" concept was derived from the "six degrees of separation" 
principle. Six degrees of separation is a theory that a person may reach anyone in 
5 the world through at the most six personal contacts. This theory was explored 
through an Internet site based upon the actor Kevin Bacon. This site was created 
out of curiosity, but was a very useful sociological experiment for illustrating the six 
degrees of separation principle. The site however, gave no hint as to any 
underlying structure of the people involved in the interconnections. 
10 The key to the underlying structure may be derived from studies into 

epidemics. For example diseases, such as the Black Death spread with great 
rapidity from one part of the world to another: from the Middle East to Europe. In 
that situation, the epidemic mechanism was that of an area that already had the 
disease within a defined group, and a single vector, such as ship-borne rats 
15 travelling to another defined group. 

By understanding the relationship between epidemics and the six degrees 
of separation principle, it is apparent that the underlying structure of the six 
degrees of separation principle was a set of richly connected groups with a small 
q number of inter-group links. It is this combination of richly connected 
20 neighborhoods, with a small number of finks between them that is termed a "small 
world" network. 

In other words, the present invention utilizes a "small-world" network 
architecture, in which a relatively small number of random cross-links of nodes or 
vertices in a network can result in small characteristic path lengths, for the transfer 
25 of messages between nodes or vertices in a telecommunications/computer 
network regardless of their location.. 
BRIEF DESCRIPTION OF THE DRAWINGS 

A preferred embodiment of the present invention will now be described, by way of 
example only, with reference to the accompanying drawings in which: - 
30 Figure 1 is a diagram of a prior art single node computer system, which has 

limited scalability; 

Figure 2 is a diagram of a prior art split node computer system; 
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Figure 3 is a diagram showing the difference between a "small world" 
network and regular ring and random networks; 

Figure 4 is a graph showing variations in clustering and average path length 
with increasing randomness in a ring network. 
5 Figure 5 is a diagram of a four-node computer system having four 

processors per node. 

Figure 6 is a diagram of a small-world architecture for a computer system in 
accordance with the invention. 
DETAILED DESCRIPTION 
10 The "small world" principle is usually considered to apply to many biological 

and social networks, as these systems generally exhibit properties that are not 
? completely regular or completely random but somewhere in between. For 
example, infectious cfiseases exhibit "small world" properties. In this regard, it has 
been shown that infectious diseases spread more easily in small world networks 
15 than regular lattices. 

The present invention applies this small world principle to 
telecommunications/computer networks. A comparison of the performance of a 
small world telecommunications network, to that of regular and random 
telecommunications networks will now be provided. Examples of regular, small- 
20 world and random networks are shown in Figure 3. 

In very large telecommunications networks with only local connections most 
vertices or nodes in the network are separated by many links. In regular lattice or 
ring structures with local connections between vertices or nodes, the characteristic 
or mean distance of the path length (L) between two vertices or nodes grows 
25 approximately linearly with the size of the network. On the other hand, in a 
network with only random connections between vertices or nodes, the 
characteristic path length (L) grows logarithmically with the number of vertices or 
nodes. A network with only random connections, however, is a poorly clustered 
environment, as a sparse number of random connections can result in some 
30 vertices or nodes not being inter-connected to other nodes at all. 

A small world telecommunications network in accordance with the present 
invention is one in which vertices or nodes in local "neighborhoods" or "clusters" 
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are interconnected to each other, and a relatively small number of random links or 
cc inections are provided between nodes of different neighborhoods of the 
network. 

In "Collective dynamics of "small-world networks" by Duncan J. Watts & 
Steven H. Strogatz, Nature, Vol. 393, pp. 440-442, a mathematical approach to 
defining structural properties of small-world networks was provided in terms of 
characteristic path length L(p) and clustering coefficient C(p). L(p) measures the 
typical separation between two vertices in a network (a global property) and C(p) 
measures the clustering or connections in a typical neighborhood (a local 
property), where p is a probability factor. 

In a regular network having n nodes and k edges or links per nodes, p = 0, 
L (0) grows linearly with n, the number of nodes in the network, and C{0) depends 
on the specific geometry or 'wiring' of the network. 

In a completely random network with only random connections, p = 1 and L- 
random grows logarithmically with n, whereas C-random ~ k/n « 1 . 

"Small-world" networks, on the other hand, have a broad interval of p over 
which L(p) is almost small as L random, yet C(p) is much greater than C-random. 
This is shown in the graph of Figure 4. 

Watts and Strogatz thus demonstrated numerically that a few random 
global connections are sufficient to turn a regular network into a small-world 
network to reduce the path length of the number of links for effective 
communication between nodes or vertices drastically. 

Hanspeter Herzel in his article entitled "How to Quantify Small-World 
Networks" (Fractals, Vol. 6, No. 4 (1998) 301-303) derived a formula for the mean 
connectivity k e * of nodes in a small-world ring network. For a ring network with k 
edges of links per node, in which a fraction of p connections are re-wired randomly 
the formula was: 

k 6 * = k . p 

When k etf > 1, the characteristic path length L(p) may be expressed as 
follows: 
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ln(n) ln(n) 
UP) = = 

IrKk 6 ") ln(k) + ln(p) 

Such a logarithmic curve approximates the curve L(p)/L(0) in Figure 4. 
5 In the present invention, the number of nodes in the network, the number of 

nodes in the neighborhood, the number of nodes per neighborhood and the 
number of cross-links in the network may each be varied for different applications, 
provided that the network functions as a small-world network with large clustering 
C(p) and small average path lengths L(p). 
10 The cross-links in the small-world network of the present invention may be 

chosen completely at random. Alternatively, a pseudo-random selection process 
h may be used to select the cross-links between neighborhoods to convert a regular 
S network into a highly clustered small-world network with a relatively small average 
O path length. In this regard, an acceptable result in terms of the number of hops 

111 

£; 15 per inter-process message, from a system process point-of-view, is under two. 

In one preferred embodiment of the invention a mean connectivity falling 

substantially in the range from about 1 .5 to about 2.0, and preferably about 1 .6, 

m may be achieved by appropriate choice of the number of nodes per neighborhood, 

S the connectivity of each neighborhood and/or the number of cross-links relative to 

N=20 the total number of nodes in the network. By way of example, in a computer 

system having 50 neighborhoods of 10 nodes arranged in a ring network with 

about 50 cross-links between neighborhoods, each node is connected to 9 other 

nodes in its neighborhood (k = 9) and the probability factor p = 50 x 2/500. Thus 

k eW = 9x 100 = 1.8 
25 500 

The number of interconnected processors in each node of the computer 
system may also vary for different applications. With the development of 
photonics, it is envisaged that up to about 256 nodes, each containing up to eight 
processors per node may be fully connected together by optical fibres and a high 
30 speed switch to form a single neighborhood in a small-world network. With a large 
number of such neighborhoods in the small-world network connected by a 
relatively small number of cross-links, it is possible to achieve a massively 



WO 01/02972 



8 



PCT/AUOO/00796 



scalable computer system with a very large number of total processors in the 
system which are able to communicate with each other in an effective manner. 

There will now be explained with reference to Figure 6, an example of how 
a "small-world" network architecture can be used to achieve a large scale 
computer system of about one million processors in which each processor of the 
system is able to communicate effectively with other processors in the system 
without requiring full connectivity. 

Figure 6 shows a small-world telecommunications network in which there 
are 20 nodes arranged in a ring layout. The 20 nodes of the ring network can be 
considered to form five neighborhoods of nodes A, B, C, D and E in which there 
are four nodes (A, to A 4 ; B, to B 4 ; to C 4 ; D, to D 4 ; and E, to E 4 ) per neighborhood. 

Each node eg ^ is connected to adjacent nodes A v A 3 by an edge link EL 
and alternate nodes eg A v A 3 ; A^ A 4 ; A 3 , B 1 are connected to each other by a small 
loop link LL. 

A network connected only in this manner would normally be termed a 
regular network in which messages from one node, eg A 1f must pass through a 
relatively large number of links to reach a node at the opposite side of the network, 
eg C 4 . 

A small-world network such as shown in Figure 6, differs from a regular 
network in that a relatively small number of random cross-links are provided 
between neighborhoods of nodes. In the example of Figure 6, there are three 
cross-links CL t , CL, and CL 3 . Cross-link CL, directly connects nodes A 2 and C 1 
with each other; cross-iink CL 2 directly connects nodes B 2 and E 3 with each other 
and cross-link CL, directly connects nodes D 1 and E 4 with each other. It will be 
apparent from Figure 6 that there is a marked increase in connectivity with only a 
relatively small number of cross-links between neighborhoods of nodes. For 
instance, a message to be sent from node A, to C 2 can pass along an edge link EL 
to node A 2 , along cross-link CL, to C, and then along an edge link to C 2 . This is a 
clear improvement over a regularly connected network, where the message would 
pass along four loop links to C, and an edge link EL to C 2 . Also, a message from 
B 4 to E 2 can pass along a loop link to B 2 , along cross-link CL, to E 4 and then along 
another loop link LL to E 2 , rather than along five loop links. 
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It will be seen from Figure 6 that a small-world network is an architecture 
that can be used to link together a relatively large number of computer processors 
while retaining effective connectivity between the processors. 

In a preferred embodiment of the invention, each neighborhood of Figure 6 
5 corresponds to the four-node/four-process per node computer system shown in 
Figure 5. Referring to Figure 5, there is shown a system that demonstrates it is 
possible to link up several MP Erlang Engines of the type shown in Figure 2 
through a high-speed blocking switch. Each node of the computer system of 
Figure 5 is an Erlang Engine 150 with four processors 152 running on a QNX 
10 kernel 156. Each Erlang Engine 150 may include a link to an OTP system 134 in 
similar manner to split node system of Figure 2 and may also have a local Internet 
connection 145. 

W The Erlang Engines 150 are connected to each other by optical interfaces 

160, optical fibres 165 and a common high speed non-blocking optical switch 170. 

; 15 Each optical interface 160 may include at least one optical receiver 161 and one 

i 

or more optical transmitters 162. 

Currently available interface blocks 160 can handle 32 separate 
wavelengths: this is predicted to rise to 256 with further developments in 
photonics. This will permit up to 256 nodes (Erlang Engines) in a computer 
20 system such as that of Figure 5, as each node can be essentially addressed by an 
individual wavelength. 

A key attribute of the system of Figure 5 is that the interface blocks can 
transmit one or more wavelengths, up to 256, with the same data simultaneously 
as a multicast facility. However, each node only receives on one wavelength: its 
25 address. 

Since the system gives almost linear scaling on separable problems, 256 
nodes can give a speed-up of 20,000 over the basic system. 

It will be noted that the computer system of Figure 5 is a fully connected 
system in the sense that each Erlang Engine 1 50 has its own dedicated optical 
30 fibre link 1 65 to the central switch 1 70. 

Applying the system of Figure 5 to the small world network of Figure 6, it is 
apparent that even where each neighborhood is a four node/four processors per 
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node system, a total of eighty processors can be effectively linked together by the 
20 node network of Figure 6 with only 3 cross-links. 

It is however, to be appreciated that the small-world network of Figure 6 is, 
however, only a relatively small-scale example of such a network. Within the 
scope of the present invention it is contemplated that at least 500 neighborhoods 
of nodes each having up to 256 fully connected Erlang Engines per node with 8 
processors per Erlang Engine could be linked together in a small-world network to 
achieve a total system of over one million processors, with the small-world network 
architecture providing effective connectivity between the nodes with only a 
relatively small number of cross-links, say 50, between the neighborhoods. For 
example, the total number of individual Erlang processors running on such a 
system will be of the order of 256,000,000 assuming 2000 active processors per 
Erlang Engine node. This gives around 25.6 million lines of code, which is of the 
order of magnitude already envisaged for large software systems project. 

It is envisaged that a massively scalable computer system in accordance 
with the invention has widespread applications. In the telecommunications field, 
the scalability could cover applications such as mobile telephone services for 
stockbroking, betting and other services. The invention may also be applied to 
Internet proxy servers in order to accommodate an escalating number of users. 

More particularly in relation to the mobile telecommunications field, the 
present invention may be used as a "personal information manager in the sky" 
whereby it coordinates communications with handheld communications devices, 
such as Personal Digital Assistants (PDAs). 

A further application is microbank applications, whereby the present 
invention may be used by financial institutions in order to allow them to 
economically handle extremely large numbers of transactions, particularly relating 
to very small amounts. 

It will be appreciated that various modifications and alterations may be 
made to the present invention as described above without departing from the 
scope and spirit of the present invention. For instance, as mentioned above, the 
size of the network, the neighborhoods in the network, the number of nodes per 
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neighborhood, the number of cross-links between neighborhoods and the number 
of processors per node may be varied for different applications. 
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THE CLAIMS DEFINING THE INVENTION ARE AS FOLLOWS: 
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1. A telecommunications/computer system having a plurality of regularly 
connected nodes, such that the plurality of nodes are clustered in a plurality 
of interconnected neighborhoods, the system characterised in that a 
relatively small number of cross-links are provided between nodes of 
different neighborhoods. 

2. The system of claim 1 wherein the relatively small number of connections 
are random links. 

3. The system of claim 1 or 2 wherein the neighborhoods are fully 
interconnected. 



4. The system of claim 1 wherein the mean connectivity between nodes of 
15 different neighborhoods is in the range of about 1 .5 to 2.0. 

5. The system of claim 1 wherein the mean connectivity between nodes of 
different neighborhoods is about 1 .6. 

20 6. A large scale computer system including a multiplicity of nodes, each node 
having a plurality of interconnected processors, said nodes being arranged 
in a network with neighboring sets of nodes of the network forming 
neighborhoods of fully interconnected nodes, wherein random links are 
provided between nodes of different neighborhoods in the network whereby 

25 each processor of the system can communicate effectively with other 

processors regardless of their location in the network without full 
connectivity in the network. 

7. A scalable computer system formed using a small world principle. 
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ATTORNEY DOCKET NO.: 27705 25 

COMBINED DECLARATION AND POWER OF ATTORNEY 



As a below named Inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name; and 

I verily believe that I am the original, first and sole inventor (if only one name is listed 
below) or an original, first and joint inventor (if plural names are listed below) of the subject 
matter which is claimed and for which a patent is sought on the invention entitled: 

A Scalable Computer System 



the specification of which: 

is attached hereto. 



/ international PCT/AU00/00796 



x was filed on Ju e ^ uuu as Application Serial No/ and was amended 

on (if applicable) (Attorney Docket No. .) 

I hereby state that I have reviewed and understand the contents of the above identified 
specification, including the claims, as amended by any amendment referred to above. 

I acknowledge the duty to disclose to the Office all information known to me to be material 
to the patentability of this application as defined in 37 CFR § 1.56. 

I hereby claim foreign priority benefits under 35 U.S.C. § 119 of any foreign 
application(s) for patent or inventor's certificate listed below and have also identified below any 
foreign application for patent or inventor's certificate having a filing date before that of any 
application on which priority is claimed: 

Country Number Date Filed Priority 

Claimed 

Australia PQ 1286 June 30, 1999 x yes 



I hereby claim the benefit under 35 U.S.C. § 120 of any United States application(s) listed 
below and, insofar as the subject matter of each of the claims of this application is not disclosed 
in the prior United States application in the manner provided by the first paragraph of 35 U.S.C, 

PDALrUOOK.l 00000-00001 
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Fergus O'Brien 


Inventor's Signature 


-/ — £ ' — o o 

10.12.2001 
Date 




Full Name 
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Melbourne. Victoria Australia. /-}Li)C^ 
Residence Citizenship Australian 






C/o, R.M.I.T., Building 91, 
Post Office Address 


3rdFloor, 110 Victoria Street, 
Melbourne. Victoria 3000. Australia. 
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Matthew Rouahan, 
Fun Name 


Inventors Signature 


11 .12. 20*^ 
Date 


■ 




North Carlton. Victoria Australia. f^Ll^s 






2 


Residence Citizenship Australian 








253 Pigdon Street, North Carlton. Victoria 3054. Australia. 
Post Office Address 






Full Name 


Inventor's Signature 


Date 
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Residence Citizenship 
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Full Name 


Inventor's Signature 


Date 
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Residence Citizenship 








Post Office Address 
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§ 1 12, 1 acknowledge the duty to disclose to the Office all information known to me to be material 
to patentability of the application as defined in 37 CFR § 1.56 which occurred between the filing 
date of the prior application and the national or PCT international filing date of this application: 

Application Serial No. Filing Date Status (patented, pending) 




I hereby appoint the following attorneys and/or agents to prosecute this application and to 
transact all business in the Patent and Trademark Office connected therewith: THOMAS L. 
CRISMAN, Reg. No. 24J846; THOMAS L. CANTRELL, Reg. No. 20349; STANLEY R. 
MOORE, Reg. No. 26,958; H. MATHEWS GARLAND, Reg. No. JS429; GERALD T. 
WELCH, Reg. No. 30,332; ROGER L. MAXWELL, Reg. No. 1LM5; P. WESTON 
MUSSELMAN, JR., Reg. No. 31,644; J. KEVIN GRAY, Reg. No. 37,14 1; JEFFERY E. 
BACON, Reg. No. 35,055; STEVEN R. GREENFIELD, Reg. No. 38466; ANDRE M. 
SZUWALSKI, Reg. No. 3531; STUART D. DWORK, Reg. No. -34x103; CRAIG A. 
HOERSTEN, Reg. No. J&,917; RICHARD J. MOURA, Reg. No. 34,883-; RICHARD L. 
MYSLIWIEC, Reg. No. 4A09&; and RAYMOND VANDYKE, Reg. No. 34.746 r of the firm 
of JENKENS & GILCHRIST, 3200 Fountain Place, 1445 Ross Avenue, Dallas, Texas 75202- 
2799; and 

Address all telephone calls and correspondence to: 

t Stanley R. Moore, Esq. 
Jenkens & Gilchrist. P.C. 
3200 Fountain Place 
1445 Ross Avenue 
Dallas, TexaT752 02-2799 
214/8S5-4500 ~* 

I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these statements 
were made with the knowledge that willful false statements and the like so made are punishable by 
fine or imprisonment, or both, under Section 1001 of Title 18 of the United States Code, and that 
such willful false statements may jeopardize the validity of the application or any patent issued 
thereon. 
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